Changing the Environment based on Intrinsic Motivation

One of the remarkable feats of intelligent life is that it restructures the world it lives in for its own benefit. Beavers build dams for shelter and to produce better hunting grounds, bees build hives for shelter and storage, humans have transformed the world in a multitude of ways. Intelligence is not only the ability to produce the right reaction to a randomly changing environment, but is also about actively influencing the change in the environment, leaving artefacts and structures that provide benefits in the future. In this abstract, I want to explore if the framework of intrinsic motivation can help us understand and possibly reproduce this phenomenon. In particular, I will show some simple, exploratory results on how empowerment, as one example of intrinsic motivation, can produce structures in the environment of an agent. The basic idea behind intrinsic motivation is that an agent’s behaviour, or decision making, is not guided by some form of externally specified reward, but rather by the optimization of some agent-internal measurement, which then gives rise to complex and rich interactions with the world. A quantitative formulation of an intrinsic motivation should ideally be computable from an agent’s perspective, be applicable to different sensory-motoric configurations, and should reflect different agent embodiments. One classic example of intrinsic motivation is Schmidhuber’s [5] artificial curiosity, where an agent acts in a way so that it learns the most about the environment. Other examples include Homeokinesis [2], or the predictive information framework [1]. Here we want to focus on empowerment [3], formally defined as the channel capacity between an agent’s actuators, and its own sensors at a later time. This measures the potential causal flow in an agent’s action-perception loop, and can be thought of as an abstract measure of how much control an agent has over the world it can perceive. The more meaningful options an agent has to influence the world, the higher is its empowerment. In a first, exploratory example, we applied discrete, 15-step empowerment to a three dimensional grid world. The model is somewhat inspired by the computer game “Minecraft” [4]. Each location is a cube, identified by its three integer coordinates. Each square can either be empty, filled with earth, or filled with an agent. Each turn the agent can take one action. The agent can, for example, decide to move in the four cardinal directions, called north, east, south and west. If the cube is empty, the agent will enter it. If the cube is filled, then there are two options. If the cube above the target cube is empty, the agent will move in its desired direction and up by one unit. If the cube above is also filled, then the agent’s move is blocked, and the agent will not move. If the agent